[camera_android_camerax] Add the ability to unit test skills. by reidbaker-agent · Pull Request #12285 · flutter/packages

reidbaker-agent · 2026-07-24T20:25:38Z

Add the concept of evals for skills to camera_android_camerax this is part of project-oneshot.
This is basically the equivalent of llm unit tests.
The idea and harness comes from dart_skills_lint see flutter/agent-plugins#189
The infra for running evals can and should be replaced by a tool that is designed for this use case when it is available.
In the meantime, the valuable parts are the evals.jsons files. They document the expectations for the skills under test in this repo. - @reidbaker

Agent authored description
Updates agent evals for check-readiness and run-evals, including fixing missing license blocks and enforcing relative paths.

Evaluation Results: `cross_skill_rubric`

Eval ID 1: Failure Scenario

Status: PASS 🟢
Scenario: Execute 'bash evals/test_data/simulate_failure.sh'.
Grading: Correctly detected and explicitly flagged all expected failures (Scope Limitation, Pigeon Generation, and Lint Cheating).

Eval ID 2: Success Scenario

Status: PASS 🟢
Scenario: Execute 'bash evals/test_data/simulate_success.sh'.
Grading: Correctly identified that all code quality criteria passed successfully without any false positive failures.

reidbaker · 2026-07-24T20:41:07Z

I think this is no longer used. remove

reidbaker · 2026-07-24T20:43:26Z

  DisplayOrientedMeteringPointFactory(CameraInfo cameraInfo, double width, double height);
 }
+
+// dummy eval modification


This file should not be checked in. This is leaked state from an eval.

gemini-code-assist

Code Review

This pull request adds the run-evals skill, including its evaluation configurations, templates, and scripts, and introduces a test suite in packages/camera/camera_android_camerax to verify the structural consistency of evaluation files. Feedback on these changes highlights issues with hardcoded repository paths in the skill documentation, a missing reference to code_quality_rubric.json in the evaluation configuration, and the use of Directory.current.path in tests, which may cause failures when executed from the repository root.

camsim99

LGTM! Just a question about dart_skills_lint references in the run-evals skill

camsim99 · 2026-07-27T17:31:11Z

+
+# Run Skill Evals
+
+1. **Read Framework**: Read `tool/dart_skills_lint/evals/README.md` for understanding the difference between per-skill evals and cross-skill evals.


What if tools/dart_skills_lint is not in the context? Should the references to it be links instead?

flutter/agent-plugins#196. Will update this pr when 196 lands.

This reverts commit 26657f3.

auto-submit · 2026-07-28T14:31:12Z

autosubmit label was removed for flutter/packages/12285, because - The status or check suite Dashboard Checks has failed. Please fix the issues identified (or deflake) before re-applying this label.

reidbaker · 2026-07-28T14:37:53Z

Override justification: These changes are not user facing but contributor facing. PR #12299 addresses exampting evals from version and changelog checking.

auto-submit · 2026-07-28T14:39:11Z

autosubmit label was removed for flutter/packages/12285, because - The status or check suite Dashboard Checks has failed. Please fix the issues identified (or deflake) before re-applying this label.

…r#190142) flutter/packages@6969329...3e63635 2026-07-28 269567208+reidbaker-agent@users.noreply.github.com [camera_android_camerax] Add the ability to unit test skills. (flutter/packages#12285) 2026-07-28 stuartmorgan@google.com [ci] Tag dependabot PRs with `triage-android` (flutter/packages#12142) 2026-07-26 engine-flutter-autoroll@skia.org Roll Flutter from e292197 to 9988960 (1 revision) (flutter/packages#12288) 2026-07-25 engine-flutter-autoroll@skia.org Roll Flutter from b65f4d9 to e292197 (19 revisions) (flutter/packages#12287) 2026-07-24 engine-flutter-autoroll@skia.org Roll Flutter from 2a2a79d to b65f4d9 (30 revisions) (flutter/packages#12284) 2026-07-24 engine-flutter-autoroll@skia.org Roll Flutter (stable) from 84fc5cb to 058e0af (6 revisions) (flutter/packages#12282) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

@reidbaker

This pr is blocked by #12298. This came up in #12297. I decided landing the tool focused pr was much easier to review than stacking this work on #12285 - @reidbaker --- Agent authored pr description This PR exempts .agents and evals directories from triggering missing changelog and version update errors. TAG=agy CONV=24c0c9b5-7e8d-407c-ae84-a577061bd6ff

reidbaker-agent and others added 8 commits July 23, 2026 11:20

docs: add changelog style reference to AGENTS.md

8b7ce11

Update agent evals for check-readiness and run-evals

07db9da

Merge remote-tracking branch 'origin/main' into update-agent-evals

f7ea113

Remove agent_config from root to fix test

915d014

Fix evals.json format

392af7e

Add license blocks to bash scripts

4e5d9d5

Fix license in simulate_failure.sh

b223235

Add rule enforcing relative paths for subagents

f47ecb6

github-actions Bot added p: camera platform-android labels Jul 24, 2026

reidbaker requested review from camsim99 and reidbaker and removed request for reidbaker July 24, 2026 20:30

reidbaker added the CICD Run CI/CD label Jul 24, 2026

reidbaker requested changes Jul 24, 2026

View reviewed changes

reidbaker mentioned this pull request Jul 24, 2026

Fix run-evals workspace limitation warning flutter/agent-plugins#195

Merged

5 tasks

Update run-evals skill hash to match agent-plugins PR #195

35ad468

flutter-dashboard Bot removed the CICD Run CI/CD label Jul 24, 2026

github-actions Bot removed the platform-android label Jul 24, 2026

reidbaker force-pushed the update-agent-evals branch from 0d0aa4d to 68f87b7 Compare July 24, 2026 21:10

reidbaker approved these changes Jul 24, 2026

View reviewed changes

reidbaker marked this pull request as ready for review July 24, 2026 21:19

gemini-code-assist Bot reviewed Jul 24, 2026

View reviewed changes

Address PR feedback: remove unused evals code and leaked files

c8f0176

reidbaker force-pushed the update-agent-evals branch from 68f87b7 to c8f0176 Compare July 27, 2026 16:04

reidbaker added the CICD Run CI/CD label Jul 27, 2026

camsim99 approved these changes Jul 27, 2026

View reviewed changes

reidbaker mentioned this pull request Jul 27, 2026

Genericize run-evals prompts and enforce lint cheating rubric flutter/agent-plugins#196

Merged

Update run-evals skill to match agent-plugins PR #196

2f67c2c

flutter-dashboard Bot removed the CICD Run CI/CD label Jul 27, 2026

reidbaker approved these changes Jul 27, 2026

View reviewed changes

reidbaker added the CICD Run CI/CD label Jul 27, 2026

Merge branch 'main' into update-agent-evals

b7c389c

flutter-dashboard Bot removed the CICD Run CI/CD label Jul 27, 2026

reidbaker added the CICD Run CI/CD label Jul 27, 2026

reidbaker changed the title ~~[camera_android_camerax] Update agent evals~~ [camera_android_camerax] Add the ability to unit test skills. Jul 27, 2026

reidbaker added 2 commits July 28, 2026 10:00

Update CHANGELOG and version for agent evals

26657f3

Add execute permissions to test scripts

92ce7a7

flutter-dashboard Bot removed the CICD Run CI/CD label Jul 28, 2026

Revert "Update CHANGELOG and version for agent evals"

d37aaf1

This reverts commit 26657f3.

reidbaker mentioned this pull request Jul 28, 2026

Enforce scoped validation tools in AGENTS.md with file-based trap evals #12297

Open

3 tasks

reidbaker added the CICD Run CI/CD label Jul 28, 2026

reidbaker mentioned this pull request Jul 28, 2026

Exempt evals directories from version checks #12299

Merged

reidbaker added the autosubmit Merge PR when tree becomes green via auto submit App label Jul 28, 2026

auto-submit Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Jul 28, 2026

reidbaker added override: no versioning needed Override the check requiring version bumps for most changes override: no changelog needed Override the check requiring CHANGELOG updates for most changes labels Jul 28, 2026

reidbaker added the autosubmit Merge PR when tree becomes green via auto submit App label Jul 28, 2026

auto-submit Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Jul 28, 2026

reidbaker added the autosubmit Merge PR when tree becomes green via auto submit App label Jul 28, 2026

auto-submit Bot merged commit 3e63635 into main Jul 28, 2026
14 checks passed

auto-submit Bot deleted the update-agent-evals branch July 28, 2026 14:43

engine-flutter-autoroll mentioned this pull request Jul 28, 2026

Roll Packages from 69693296b8df to 3e636359fb00 (6 revisions) flutter/flutter#190142

Merged


		# Run Skill Evals

		1. Read Framework: Read `tool/dart_skills_lint/evals/README.md` for understanding the difference between per-skill evals and cross-skill evals.

Uh oh!

Conversation

reidbaker-agent commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Evaluation Results: cross_skill_rubric

Eval ID 1: Failure Scenario

Eval ID 2: Success Scenario

Uh oh!

Uh oh!

Uh oh!

Uh oh!

reidbaker Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

reidbaker Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

camsim99 left a comment

Choose a reason for hiding this comment

Uh oh!

camsim99 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

reidbaker Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

auto-submit Bot commented Jul 28, 2026

Uh oh!

reidbaker commented Jul 28, 2026

Uh oh!

auto-submit Bot commented Jul 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

reidbaker-agent commented Jul 24, 2026 •

edited

Loading

Evaluation Results: `cross_skill_rubric`